{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6x1ypzczQCwy"
   },
   "source": [
    "# Creating a Simple TFX Pipeline for Vertex AI Pipelines\n",
    "\n",
    "## Learning objectives\n",
    "\n",
    "1. Prepare example data.\n",
    "2. Create a pipeline.\n",
    "3. Run the pipeline on Vertex AI Pipelines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_VuwrlnvQJ5k"
   },
   "source": [
    "## Introduction\n",
    "In this notebook, you will create a simple TFX pipeline and run it using\n",
    "Google Cloud Vertex Pipelines.  This notebook is based on the TFX pipeline\n",
    "you built in\n",
    "[Simple TFX Pipeline Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple).\n",
    "If you are not familiar with TFX and you have not read that tutorial yet, you\n",
    "should read it before proceeding with this notebook.\n",
    "\n",
    "Google Cloud Vertex Pipelines helps you to automate, monitor, and govern\n",
    "your ML systems by orchestrating your ML workflow in a serverless manner. You\n",
    "can define your ML pipelines using Python with TFX, and then execute your\n",
    "pipelines on Google Cloud. See\n",
    "[Vertex Pipelines introduction](https://cloud.google.com/vertex-ai/docs/pipelines/introduction)\n",
    "to learn more about Vertex Pipelines.\n",
    "\n",
    "Each learning objective will correspond to a _#TODO_ in this student lab notebook -- try to complete this notebook first and then review the [solution notebook](../solutions/Simple_TFX_Pipeline_for_Vertex_Pipelines.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fwZ0aXisoBFW"
   },
   "source": [
    "### Install python packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WC9W_S-bONgl"
   },
   "source": [
    "You will install required Python packages including TFX and KFP to author ML\n",
    "pipelines and submit jobs to Vertex Pipelines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "iyQtljP-qPHY"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pip in /opt/conda/lib/python3.7/site-packages (22.0.4)\n",
      "Collecting tfx[kfp]<2\n",
      "  Downloading tfx-1.7.1-py3-none-any.whl (2.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.5/2.5 MB\u001b[0m \u001b[31m27.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: tfx-bsl<1.8.0,>=1.7.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.7.0)\n",
      "Collecting packaging<21,>=20\n",
      "  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.9/40.9 KB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-apitools<1,>=0.5 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (0.5.31)\n",
      "Requirement already satisfied: tensorflow-serving-api!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3,>=1.15 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.8.0)\n",
      "Requirement already satisfied: keras-tuner<2,>=1.0.4 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.1.0)\n",
      "Requirement already satisfied: apache-beam[gcp]<3,>=2.36 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.37.0)\n",
      "Collecting google-api-python-client<2,>=1.8\n",
      "  Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl (62 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.1/62.1 KB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-cloud-aiplatform<2,>=1.6.2 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.11.0)\n",
      "Requirement already satisfied: numpy<2,>=1.16 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.19.5)\n",
      "Collecting tensorflow-model-analysis<0.39,>=0.38.0\n",
      "  Downloading tensorflow_model_analysis-0.38.0-py3-none-any.whl (1.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m58.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting pyarrow<6,>=1\n",
      "  Downloading pyarrow-5.0.0-cp37-cp37m-manylinux2014_x86_64.whl (23.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m23.6/23.6 MB\u001b[0m \u001b[31m45.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting click<8,>=7\n",
      "  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m82.8/82.8 KB\u001b[0m \u001b[31m13.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting tensorflow-data-validation<1.8.0,>=1.7.0\n",
      "  Downloading tensorflow_data_validation-1.7.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m68.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: grpcio<2,>=1.28.1 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.44.0)\n",
      "Collecting ml-pipelines-sdk==1.7.1\n",
      "  Downloading ml_pipelines_sdk-1.7.1-py3-none-any.whl (1.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m55.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: protobuf<4,>=3.13 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (3.19.4)\n",
      "Requirement already satisfied: google-cloud-bigquery<3,>=2.26.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.34.2)\n",
      "Collecting docker<5,>=4.1\n",
      "  Downloading docker-4.4.4-py2.py3-none-any.whl (147 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m147.0/147.0 KB\u001b[0m \u001b[31m21.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: tensorflow-transform<1.8.0,>=1.7.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.7.0)\n",
      "Requirement already satisfied: jinja2<4,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.11.3)\n",
      "Collecting pyyaml<6,>=3.12\n",
      "  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m636.6/636.6 KB\u001b[0m \u001b[31m49.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: absl-py<2.0.0,>=0.9 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (0.15.0)\n",
      "Collecting portpicker<2,>=1.3.1\n",
      "  Downloading portpicker-1.5.0-py3-none-any.whl (14 kB)\n",
      "Collecting attrs<21,>=19.3.0\n",
      "  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.3/49.3 KB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: tensorflow-hub<0.13,>=0.9.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (0.12.0)\n",
      "Collecting ml-metadata<1.8.0,>=1.7.0\n",
      "  Downloading ml_metadata-1.7.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.6/6.6 MB\u001b[0m \u001b[31m60.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0mm\n",
      "\u001b[?25hCollecting kubernetes<13,>=10.0.1\n",
      "  Downloading kubernetes-12.0.1-py2.py3-none-any.whl (1.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m79.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5\n",
      "  Downloading tensorflow-2.8.0-cp37-cp37m-manylinux2010_x86_64.whl (497.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m497.5/497.5 MB\u001b[0m \u001b[31m1.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting kfp-pipeline-spec<0.2,>=0.1.10\n",
      "  Downloading kfp_pipeline_spec-0.1.14-py3-none-any.whl (18 kB)\n",
      "Collecting kfp<2,>=1.8.5\n",
      "  Downloading kfp-1.8.12.tar.gz (301 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m301.2/301.2 KB\u001b[0m \u001b[31m32.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from absl-py<2.0.0,>=0.9->tfx[kfp]<2) (1.15.0)\n",
      "Collecting httplib2<0.20.0,>=0.8\n",
      "  Downloading httplib2-0.19.1-py3-none-any.whl (95 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m95.4/95.4 KB\u001b[0m \u001b[31m12.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: pytz>=2018.3 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2021.3)\n",
      "Requirement already satisfied: proto-plus<2,>=1.7.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.20.3)\n",
      "Requirement already satisfied: fastavro<2,>=0.23.6 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.4.10)\n",
      "Requirement already satisfied: pydot<2,>=1.2.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.4.2)\n",
      "Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.6.0)\n",
      "Requirement already satisfied: pymongo<4.0.0,>=3.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.12.3)\n",
      "Requirement already satisfied: typing-extensions>=3.7.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.10.0.2)\n",
      "Requirement already satisfied: crcmod<2.0,>=1.7 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.7)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.24.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.27.1)\n",
      "Requirement already satisfied: python-dateutil<3,>=2.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.8.2)\n",
      "Collecting dill<0.3.2,>=0.3.1.1\n",
      "  Downloading dill-0.3.1.1.tar.gz (151 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m152.0/152.0 KB\u001b[0m \u001b[31m24.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: cloudpickle<3,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.0.0)\n",
      "Requirement already satisfied: orjson<4.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.6.7)\n",
      "Requirement already satisfied: oauth2client<5,>=2.0.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (4.1.3)\n",
      "Collecting google-cloud-videointelligence<2,>=1.8.0\n",
      "  Downloading google_cloud_videointelligence-1.16.2-py2.py3-none-any.whl (183 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m183.9/183.9 KB\u001b[0m \u001b[31m26.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting google-cloud-spanner<2,>=1.13.0\n",
      "  Downloading google_cloud_spanner-1.19.2-py2.py3-none-any.whl (255 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m255.5/255.5 KB\u001b[0m \u001b[31m32.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: cachetools<5,>=3.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (4.2.4)\n",
      "Requirement already satisfied: google-auth<3,>=1.18.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.35.0)\n",
      "Requirement already satisfied: google-cloud-dlp<4,>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.6.2)\n",
      "Collecting google-cloud-datastore<2,>=1.8.0\n",
      "  Downloading google_cloud_datastore-1.15.4-py2.py3-none-any.whl (134 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.2/134.2 KB\u001b[0m \u001b[31m20.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting google-cloud-language<2,>=1.3.0\n",
      "  Downloading google_cloud_language-1.3.1-py2.py3-none-any.whl (83 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m83.6/83.6 KB\u001b[0m \u001b[31m11.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: grpcio-gcp<1,>=0.2.2 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.2.2)\n",
      "Requirement already satisfied: google-cloud-pubsub<3,>=2.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.11.0)\n",
      "Collecting google-cloud-bigtable<2,>=0.31.1\n",
      "  Downloading google_cloud_bigtable-1.7.1-py2.py3-none-any.whl (267 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m267.7/267.7 KB\u001b[0m \u001b[31m28.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-cloud-bigquery-storage>=2.6.3 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.13.0)\n",
      "Requirement already satisfied: google-cloud-pubsublite<2,>=1.2.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.4.1)\n",
      "Collecting google-cloud-vision<2,>=0.38.0\n",
      "  Downloading google_cloud_vision-1.0.1-py2.py3-none-any.whl (435 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m435.1/435.1 KB\u001b[0m \u001b[31m47.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting google-cloud-core<2,>=0.28.1\n",
      "  Downloading google_cloud_core-1.7.2-py2.py3-none-any.whl (28 kB)\n",
      "Requirement already satisfied: google-cloud-recommendations-ai<=0.2.0,>=0.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.2.0)\n",
      "Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker<5,>=4.1->tfx[kfp]<2) (1.3.1)\n",
      "Collecting uritemplate<4dev,>=3.0.0\n",
      "  Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)\n",
      "Requirement already satisfied: google-api-core<3dev,>=1.21.0 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.8->tfx[kfp]<2) (2.5.0)\n",
      "Requirement already satisfied: google-auth-httplib2>=0.0.3 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.8->tfx[kfp]<2) (0.1.0)\n",
      "Requirement already satisfied: fasteners>=0.14 in /opt/conda/lib/python3.7/site-packages (from google-apitools<1,>=0.5->tfx[kfp]<2) (0.17.3)\n",
      "Requirement already satisfied: google-cloud-storage<3.0.0dev,>=1.32.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx[kfp]<2) (2.2.1)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (2.3.2)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from jinja2<4,>=2.7.3->tfx[kfp]<2) (2.0.1)\n",
      "Requirement already satisfied: scipy in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (1.7.3)\n",
      "Requirement already satisfied: tensorboard in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (2.6.0)\n",
      "Requirement already satisfied: ipython in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (7.32.0)\n",
      "Requirement already satisfied: kt-legacy in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (1.0.4)\n",
      "Collecting google-cloud-storage<3.0.0dev,>=1.32.0\n",
      "  Downloading google_cloud_storage-1.44.0-py2.py3-none-any.whl (106 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m106.8/106.8 KB\u001b[0m \u001b[31m15.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting requests-toolbelt<1,>=0.8.0\n",
      "  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.3/54.3 KB\u001b[0m \u001b[31m8.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting kfp-server-api<2.0.0,>=1.1.2\n",
      "  Downloading kfp-server-api-1.8.1.tar.gz (54 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.2/54.2 KB\u001b[0m \u001b[31m9.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting jsonschema<4,>=3.0.1\n",
      "  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.3/56.3 KB\u001b[0m \u001b[31m10.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting tabulate<1,>=0.8.6\n",
      "  Downloading tabulate-0.8.9-py3-none-any.whl (25 kB)\n",
      "Collecting Deprecated<2,>=1.2.7\n",
      "  Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)\n",
      "Collecting strip-hints<1,>=0.1.8\n",
      "  Downloading strip-hints-0.1.10.tar.gz (29 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting docstring-parser<1,>=0.7.3\n",
      "  Downloading docstring_parser-0.14.1-py3-none-any.whl (33 kB)\n",
      "Collecting fire<1,>=0.3.1\n",
      "  Downloading fire-0.4.0.tar.gz (87 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m87.7/87.7 KB\u001b[0m \u001b[31m415.4 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: pydantic<2,>=1.8.2 in /opt/conda/lib/python3.7/site-packages (from kfp<2,>=1.8.5->tfx[kfp]<2) (1.9.0)\n",
      "Collecting typer<1.0,>=0.3.2\n",
      "  Downloading typer-0.4.1-py3-none-any.whl (27 kB)\n",
      "Requirement already satisfied: certifi>=14.05.14 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (2021.10.8)\n",
      "Requirement already satisfied: setuptools>=21.0.0 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (59.8.0)\n",
      "Requirement already satisfied: urllib3>=1.24.2 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (1.26.8)\n",
      "Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (1.3.1)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging<21,>=20->tfx[kfp]<2) (3.0.7)\n",
      "Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from portpicker<2,>=1.3.1->tfx[kfp]<2) (5.9.0)\n",
      "Collecting libclang>=9.0.1\n",
      "  Downloading libclang-14.0.1-py2.py3-none-manylinux1_x86_64.whl (14.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.5/14.5 MB\u001b[0m \u001b[31m71.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: keras-preprocessing>=1.1.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.1.2)\n",
      "Collecting tf-estimator-nightly==2.8.0.dev2021122109\n",
      "  Downloading tf_estimator_nightly-2.8.0.dev2021122109-py2.py3-none-any.whl (462 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m462.5/462.5 KB\u001b[0m \u001b[31m41.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting numpy<2,>=1.16\n",
      "  Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.7/15.7 MB\u001b[0m \u001b[31m70.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting tensorboard\n",
      "  Downloading tensorboard-2.8.0-py3-none-any.whl (5.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.8/5.8 MB\u001b[0m \u001b[31m96.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: h5py>=2.9.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (3.1.0)\n",
      "Requirement already satisfied: astunparse>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.6.3)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.1.0)\n",
      "Collecting keras<2.9,>=2.8.0rc0\n",
      "  Downloading keras-2.8.0-py2.py3-none-any.whl (1.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m62.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting tensorflow-io-gcs-filesystem>=0.23.1\n",
      "  Downloading tensorflow_io_gcs_filesystem-0.25.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m82.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: gast>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (0.4.0)\n",
      "Requirement already satisfied: flatbuffers>=1.12 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.12)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (3.3.0)\n",
      "Requirement already satisfied: wrapt>=1.11.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.12.1)\n",
      "Requirement already satisfied: google-pasta>=0.1.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (0.2.0)\n",
      "Collecting joblib<0.15,>=0.12\n",
      "  Downloading joblib-0.14.1-py2.py3-none-any.whl (294 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m294.9/294.9 KB\u001b[0m \u001b[31m34.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting pyfarmhash<0.4,>=0.2\n",
      "  Downloading pyfarmhash-0.3.2.tar.gz (99 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m99.9/99.9 KB\u001b[0m \u001b[31m15.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: pandas<2,>=1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-data-validation<1.8.0,>=1.7.0->tfx[kfp]<2) (1.3.5)\n",
      "Requirement already satisfied: tensorflow-metadata<1.8,>=1.7.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-data-validation<1.8.0,>=1.7.0->tfx[kfp]<2) (1.7.0)\n",
      "Requirement already satisfied: ipywidgets<8,>=7 in /opt/conda/lib/python3.7/site-packages (from tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (7.6.5)\n",
      "Requirement already satisfied: wheel<1.0,>=0.23.0 in /opt/conda/lib/python3.7/site-packages (from astunparse>=1.6.0->tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (0.37.1)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.52.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client<2,>=1.8->tfx[kfp]<2) (1.54.0)\n",
      "Requirement already satisfied: grpcio-status<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client<2,>=1.8->tfx[kfp]<2) (1.44.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (4.8)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.2.7)\n",
      "Requirement already satisfied: grpc-google-iam-v1<0.13dev,>=0.12.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigtable<2,>=0.31.1->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.12.3)\n",
      "Collecting google-cloud-core<2,>=0.28.1\n",
      "  Downloading google_cloud_core-1.7.1-py2.py3-none-any.whl (28 kB)\n",
      "  Downloading google_cloud_core-1.7.0-py2.py3-none-any.whl (28 kB)\n",
      "  Downloading google_cloud_core-1.6.0-py2.py3-none-any.whl (28 kB)\n",
      "  Downloading google_cloud_core-1.5.0-py2.py3-none-any.whl (27 kB)\n",
      "  Downloading google_cloud_core-1.4.4-py2.py3-none-any.whl (27 kB)\n",
      "  Downloading google_cloud_core-1.4.3-py2.py3-none-any.whl (27 kB)\n",
      "  Downloading google_cloud_core-1.4.2-py2.py3-none-any.whl (26 kB)\n",
      "  Downloading google_cloud_core-1.4.1-py2.py3-none-any.whl (26 kB)\n",
      "INFO: pip is looking at multiple versions of google-cloud-bigtable to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-cloud-bigtable<2,>=0.31.1\n",
      "  Downloading google_cloud_bigtable-1.7.0-py2.py3-none-any.whl (267 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m267.7/267.7 KB\u001b[0m \u001b[31m35.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-cloud-bigquery-storage to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-cloud-bigquery-storage>=2.6.3\n",
      "  Downloading google_cloud_bigquery_storage-2.13.1-py2.py3-none-any.whl (180 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m180.1/180.1 KB\u001b[0m \u001b[31m27.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-auth-httplib2 to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-auth-httplib2>=0.0.3\n",
      "  Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)\n",
      "INFO: pip is looking at multiple versions of google-auth to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-auth<3,>=1.18.0\n",
      "  Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m152.9/152.9 KB\u001b[0m \u001b[31m21.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5\n",
      "  Downloading google_api_core-2.7.3-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 KB\u001b[0m \u001b[31m19.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core to determine which version is compatible with other requirements. This could take a while.\n",
      "  Downloading google_api_core-2.7.2-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 KB\u001b[0m \u001b[31m19.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.7.1-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.7/114.7 KB\u001b[0m \u001b[31m17.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.7.0-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.7/114.7 KB\u001b[0m \u001b[31m17.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.6.1-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 KB\u001b[0m \u001b[31m17.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 KB\u001b[0m \u001b[31m18.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.8/111.8 KB\u001b[0m \u001b[31m11.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.\n",
      "  Downloading google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.7/111.7 KB\u001b[0m \u001b[31m15.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core to determine which version is compatible with other requirements. This could take a while.\n",
      "  Downloading google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m109.8/109.8 KB\u001b[0m \u001b[31m15.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m93.3/93.3 KB\u001b[0m \u001b[31m14.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: overrides<7.0.0,>=6.0.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (6.1.0)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (1.1.2)\n",
      "Requirement already satisfied: cached-property in /opt/conda/lib/python3.7/site-packages (from h5py>=2.9.0->tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.5.2)\n",
      "Requirement already satisfied: docopt in /opt/conda/lib/python3.7/site-packages (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.6.2)\n",
      "Collecting pyparsing>=2.0.2\n",
      "  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.8/67.8 KB\u001b[0m \u001b[31m10.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: backcall in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.2.0)\n",
      "Requirement already satisfied: decorator in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (5.1.1)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (3.0.27)\n",
      "Requirement already satisfied: pygments in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (2.11.2)\n",
      "Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.18.1)\n",
      "Requirement already satisfied: matplotlib-inline in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.1.3)\n",
      "Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (4.8.0)\n",
      "Requirement already satisfied: traitlets>=4.2 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (5.1.1)\n",
      "Requirement already satisfied: pickleshare in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.7.5)\n",
      "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.0.2)\n",
      "Requirement already satisfied: nbformat>=4.2.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (5.2.0)\n",
      "Requirement already satisfied: ipython-genutils~=0.2.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.2.0)\n",
      "Requirement already satisfied: widgetsnbextension~=3.5.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (3.5.2)\n",
      "Requirement already satisfied: ipykernel>=4.5.1 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (6.9.2)\n",
      "Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp<2,>=1.8.5->tfx[kfp]<2) (4.11.3)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp<2,>=1.8.5->tfx[kfp]<2) (0.18.1)\n",
      "Requirement already satisfied: pyasn1>=0.1.7 in /opt/conda/lib/python3.7/site-packages (from oauth2client<5,>=2.0.1->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.4.8)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.3)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.0.12)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.4.6)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (3.3.6)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (1.8.1)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (2.0.3)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.6.1)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib->kubernetes<13,>=10.0.1->tfx[kfp]<2) (3.2.0)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (1.15.0)\n",
      "Requirement already satisfied: debugpy<2.0,>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.5.1)\n",
      "Requirement already satisfied: tornado<7.0,>=4.2 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (6.1)\n",
      "Requirement already satisfied: jupyter-client<8.0 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (7.1.2)\n",
      "Requirement already satisfied: nest-asyncio in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.5.4)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.7/site-packages (from jedi>=0.16->ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.8.3)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->jsonschema<4,>=3.0.1->kfp<2,>=1.8.5->tfx[kfp]<2) (3.7.0)\n",
      "Requirement already satisfied: jupyter-core in /opt/conda/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (4.9.2)\n",
      "Requirement already satisfied: typing-utils>=0.0.3 in /opt/conda/lib/python3.7/site-packages (from overrides<7.0.0,>=6.0.1->google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.1.0)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.7/site-packages (from pexpect>4.3->ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.7.0)\n",
      "Requirement already satisfied: wcwidth in /opt/conda/lib/python3.7/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.2.5)\n",
      "Requirement already satisfied: notebook>=4.4.1 in /opt/conda/lib/python3.7/site-packages (from widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (6.4.10)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (2.21)\n",
      "Requirement already satisfied: entrypoints in /opt/conda/lib/python3.7/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.4)\n",
      "Requirement already satisfied: pyzmq>=13 in /opt/conda/lib/python3.7/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (22.3.0)\n",
      "Requirement already satisfied: nbconvert>=5 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (6.4.4)\n",
      "Requirement already satisfied: argon2-cffi in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (21.3.0)\n",
      "Requirement already satisfied: terminado>=0.8.3 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.13.3)\n",
      "Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.13.1)\n",
      "Requirement already satisfied: Send2Trash>=1.8.0 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.8.0)\n",
      "Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.5.13)\n",
      "Requirement already satisfied: mistune<2,>=0.8.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.8.4)\n",
      "Requirement already satisfied: beautifulsoup4 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (4.10.0)\n",
      "Requirement already satisfied: defusedxml in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.7.1)\n",
      "Requirement already satisfied: bleach in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (4.1.0)\n",
      "Requirement already satisfied: testpath in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.6.0)\n",
      "Requirement already satisfied: jupyterlab-pygments in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.1.2)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.5.0)\n",
      "Requirement already satisfied: argon2-cffi-bindings in /opt/conda/lib/python3.7/site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (21.2.0)\n",
      "Requirement already satisfied: soupsieve>1.2 in /opt/conda/lib/python3.7/site-packages (from beautifulsoup4->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (2.3.1)\n",
      "Requirement already satisfied: webencodings in /opt/conda/lib/python3.7/site-packages (from bleach->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.5.1)\n",
      "Building wheels for collected packages: kfp, dill, fire, kfp-server-api, pyfarmhash, strip-hints\n",
      "  Building wheel for kfp (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for kfp: filename=kfp-1.8.12-py3-none-any.whl size=419048 sha256=fc5fc7f3a9b5f676cd53e5cd69e492ed3e995211cc4b6b8cfc5a06b54796f1d4\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/54/0c/4a/3fc55077bc88cc17eacaae34c5fd3f6178c1d16d2ee3b0afdf\n",
      "  Building wheel for dill (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for dill: filename=dill-0.3.1.1-py3-none-any.whl size=78544 sha256=6d34132d6cad3e94afaaf4057513f0bcf6eec4626ab3183f05c6bb5e4d8f93bf\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/a4/61/fd/c57e374e580aa78a45ed78d5859b3a44436af17e22ca53284f\n",
      "  Building wheel for fire (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for fire: filename=fire-0.4.0-py2.py3-none-any.whl size=115942 sha256=a0e96813de3269a653f0bbe1d106b94a930df65b64cfc8c8afd4091e2af841b1\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/8a/67/fb/2e8a12fa16661b9d5af1f654bd199366799740a85c64981226\n",
      "  Building wheel for kfp-server-api (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for kfp-server-api: filename=kfp_server_api-1.8.1-py3-none-any.whl size=95549 sha256=1d5907bdf9958cf5fd1529a0fa69ee8a91fefca39b8c3942766a2365d59478f3\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/f5/4e/2e/6795bd3ed456a43652e7de100aca275ec179c9a8dfbcc65626\n",
      "  Building wheel for pyfarmhash (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for pyfarmhash: filename=pyfarmhash-0.3.2-cp37-cp37m-linux_x86_64.whl size=108625 sha256=1ba31109887fb167782b9678b72ce85552e486cea10a26e776ef9b5d238c512d\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/53/58/7a/3b040f3a2ee31908e3be916e32660db6db53621ce6eba838dc\n",
      "  Building wheel for strip-hints (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for strip-hints: filename=strip_hints-0.1.10-py2.py3-none-any.whl size=22302 sha256=1fa632e6899f84956f31c089308aec5d16a45626c95b97bbb9dd78e5851779bb\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/5e/14/c3/6e44e9b2545f2d570b03f5b6d38c00b7534aa8abb376978363\n",
      "Successfully built kfp dill fire kfp-server-api pyfarmhash strip-hints\n",
      "Installing collected packages: tf-estimator-nightly, tabulate, pyfarmhash, libclang, keras, joblib, uritemplate, tensorflow-io-gcs-filesystem, strip-hints, pyyaml, pyparsing, portpicker, numpy, kfp-pipeline-spec, fire, docstring-parser, dill, Deprecated, click, attrs, typer, requests-toolbelt, pyarrow, packaging, ml-metadata, kfp-server-api, jsonschema, httplib2, docker, kubernetes, google-api-core, tensorboard, google-cloud-core, google-api-python-client, tensorflow, ml-pipelines-sdk, google-cloud-vision, google-cloud-videointelligence, google-cloud-storage, google-cloud-spanner, google-cloud-language, google-cloud-datastore, google-cloud-bigtable, kfp, tensorflow-model-analysis, tensorflow-data-validation, tfx\n",
      "  Attempting uninstall: keras\n",
      "    Found existing installation: keras 2.6.0\n",
      "    Uninstalling keras-2.6.0:\n",
      "      Successfully uninstalled keras-2.6.0\n",
      "  Attempting uninstall: joblib\n",
      "    Found existing installation: joblib 1.0.1\n",
      "    Uninstalling joblib-1.0.1:\n",
      "      Successfully uninstalled joblib-1.0.1\n",
      "  Attempting uninstall: uritemplate\n",
      "    Found existing installation: uritemplate 4.1.1\n",
      "    Uninstalling uritemplate-4.1.1:\n",
      "      Successfully uninstalled uritemplate-4.1.1\n",
      "  Attempting uninstall: pyyaml\n",
      "    Found existing installation: PyYAML 6.0\n",
      "    Uninstalling PyYAML-6.0:\n",
      "      Successfully uninstalled PyYAML-6.0\n",
      "  Attempting uninstall: pyparsing\n",
      "    Found existing installation: pyparsing 3.0.7\n",
      "    Uninstalling pyparsing-3.0.7:\n",
      "      Successfully uninstalled pyparsing-3.0.7\n",
      "  Attempting uninstall: numpy\n",
      "    Found existing installation: numpy 1.19.5\n",
      "    Uninstalling numpy-1.19.5:\n",
      "      Successfully uninstalled numpy-1.19.5\n",
      "  Attempting uninstall: dill\n",
      "    Found existing installation: dill 0.3.4\n",
      "    Uninstalling dill-0.3.4:\n",
      "      Successfully uninstalled dill-0.3.4\n",
      "  Attempting uninstall: click\n",
      "    Found existing installation: click 8.0.4\n",
      "    Uninstalling click-8.0.4:\n",
      "      Successfully uninstalled click-8.0.4\n",
      "  Attempting uninstall: attrs\n",
      "    Found existing installation: attrs 21.4.0\n",
      "    Uninstalling attrs-21.4.0:\n",
      "      Successfully uninstalled attrs-21.4.0\n",
      "  Attempting uninstall: pyarrow\n",
      "    Found existing installation: pyarrow 7.0.0\n",
      "    Uninstalling pyarrow-7.0.0:\n",
      "      Successfully uninstalled pyarrow-7.0.0\n",
      "  Attempting uninstall: packaging\n",
      "    Found existing installation: packaging 21.3\n",
      "    Uninstalling packaging-21.3:\n",
      "      Successfully uninstalled packaging-21.3\n",
      "  Attempting uninstall: jsonschema\n",
      "    Found existing installation: jsonschema 4.4.0\n",
      "    Uninstalling jsonschema-4.4.0:\n",
      "      Successfully uninstalled jsonschema-4.4.0\n",
      "  Attempting uninstall: httplib2\n",
      "    Found existing installation: httplib2 0.20.4\n",
      "    Uninstalling httplib2-0.20.4:\n",
      "      Successfully uninstalled httplib2-0.20.4\n",
      "  Attempting uninstall: docker\n",
      "    Found existing installation: docker 5.0.3\n",
      "    Uninstalling docker-5.0.3:\n",
      "      Successfully uninstalled docker-5.0.3\n",
      "  Attempting uninstall: kubernetes\n",
      "    Found existing installation: kubernetes 23.3.0\n",
      "    Uninstalling kubernetes-23.3.0:\n",
      "      Successfully uninstalled kubernetes-23.3.0\n",
      "  Attempting uninstall: google-api-core\n",
      "    Found existing installation: google-api-core 2.5.0\n",
      "    Uninstalling google-api-core-2.5.0:\n",
      "      Successfully uninstalled google-api-core-2.5.0\n",
      "  Attempting uninstall: tensorboard\n",
      "    Found existing installation: tensorboard 2.6.0\n",
      "    Uninstalling tensorboard-2.6.0:\n",
      "      Successfully uninstalled tensorboard-2.6.0\n",
      "  Attempting uninstall: google-cloud-core\n",
      "    Found existing installation: google-cloud-core 2.2.3\n",
      "    Uninstalling google-cloud-core-2.2.3:\n",
      "      Successfully uninstalled google-cloud-core-2.2.3\n",
      "  Attempting uninstall: google-api-python-client\n",
      "    Found existing installation: google-api-python-client 2.41.0\n",
      "    Uninstalling google-api-python-client-2.41.0:\n",
      "      Successfully uninstalled google-api-python-client-2.41.0\n",
      "  Attempting uninstall: tensorflow\n",
      "    Found existing installation: tensorflow 2.6.3\n",
      "    Uninstalling tensorflow-2.6.3:\n",
      "      Successfully uninstalled tensorflow-2.6.3\n",
      "  Attempting uninstall: google-cloud-vision\n",
      "    Found existing installation: google-cloud-vision 2.7.1\n",
      "    Uninstalling google-cloud-vision-2.7.1:\n",
      "      Successfully uninstalled google-cloud-vision-2.7.1\n",
      "  Attempting uninstall: google-cloud-videointelligence\n",
      "    Found existing installation: google-cloud-videointelligence 2.6.1\n",
      "    Uninstalling google-cloud-videointelligence-2.6.1:\n",
      "      Successfully uninstalled google-cloud-videointelligence-2.6.1\n",
      "  Attempting uninstall: google-cloud-storage\n",
      "    Found existing installation: google-cloud-storage 2.2.1\n",
      "    Uninstalling google-cloud-storage-2.2.1:\n",
      "      Successfully uninstalled google-cloud-storage-2.2.1\n",
      "  Attempting uninstall: google-cloud-spanner\n",
      "    Found existing installation: google-cloud-spanner 3.13.0\n",
      "    Uninstalling google-cloud-spanner-3.13.0:\n",
      "      Successfully uninstalled google-cloud-spanner-3.13.0\n",
      "  Attempting uninstall: google-cloud-language\n",
      "    Found existing installation: google-cloud-language 2.4.1\n",
      "    Uninstalling google-cloud-language-2.4.1:\n",
      "      Successfully uninstalled google-cloud-language-2.4.1\n",
      "  Attempting uninstall: google-cloud-datastore\n",
      "    Found existing installation: google-cloud-datastore 2.5.1\n",
      "    Uninstalling google-cloud-datastore-2.5.1:\n",
      "      Successfully uninstalled google-cloud-datastore-2.5.1\n",
      "  Attempting uninstall: google-cloud-bigtable\n",
      "    Found existing installation: google-cloud-bigtable 2.7.0\n",
      "    Uninstalling google-cloud-bigtable-2.7.0:\n",
      "      Successfully uninstalled google-cloud-bigtable-2.7.0\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "tensorflow-io 0.21.0 requires tensorflow<2.7.0,>=2.6.0, but you have tensorflow 2.8.0 which is incompatible.\n",
      "tensorflow-io 0.21.0 requires tensorflow-io-gcs-filesystem==0.21.0, but you have tensorflow-io-gcs-filesystem 0.25.0 which is incompatible.\n",
      "statsmodels 0.13.2 requires packaging>=21.3, but you have packaging 20.9 which is incompatible.\n",
      "pandas-profiling 3.1.0 requires joblib~=1.0.1, but you have joblib 0.14.1 which is incompatible.\n",
      "cloud-tpu-client 0.10 requires google-api-python-client==1.8.0, but you have google-api-python-client 1.12.11 which is incompatible.\n",
      "black 22.1.0 requires click>=8.0.0, but you have click 7.1.2 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed Deprecated-1.2.13 attrs-20.3.0 click-7.1.2 dill-0.3.1.1 docker-4.4.4 docstring-parser-0.14.1 fire-0.4.0 google-api-core-1.31.5 google-api-python-client-1.12.11 google-cloud-bigtable-1.7.1 google-cloud-core-2.2.2 google-cloud-datastore-1.15.4 google-cloud-language-1.3.1 google-cloud-spanner-1.19.2 google-cloud-storage-2.1.0 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 httplib2-0.19.1 joblib-0.14.1 jsonschema-3.2.0 keras-2.8.0 kfp-1.8.12 kfp-pipeline-spec-0.1.14 kfp-server-api-1.8.1 kubernetes-12.0.1 libclang-14.0.1 ml-metadata-1.7.0 ml-pipelines-sdk-1.7.1 numpy-1.21.6 packaging-20.9 portpicker-1.5.0 pyarrow-5.0.0 pyfarmhash-0.3.2 pyparsing-2.4.7 pyyaml-5.4.1 requests-toolbelt-0.9.1 strip-hints-0.1.10 tabulate-0.8.9 tensorboard-2.8.0 tensorflow-2.8.0 tensorflow-data-validation-1.7.0 tensorflow-io-gcs-filesystem-0.25.0 tensorflow-model-analysis-0.38.0 tf-estimator-nightly-2.8.0.dev2021122109 tfx-1.7.1 typer-0.4.1 uritemplate-3.0.1\n"
     ]
    }
   ],
   "source": [
    "# Use the latest version of pip.\n",
    "!pip install --upgrade pip\n",
    "!pip install --upgrade \"tfx[kfp]<2\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EwT0nov5QO1M"
   },
   "source": [
    "#### Did you restart the runtime?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-CRyIL4LVDlQ"
   },
   "source": [
    "You can restart runtime with following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "KHTSzMygoBF6"
   },
   "outputs": [],
   "source": [
    "# docs_infra: no_execute\n",
    "import sys\n",
    "if not 'google.colab' in sys.modules:\n",
    "  # Automatically restart kernel after installs\n",
    "  import IPython\n",
    "  app = IPython.Application.instance()\n",
    "  app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3_SveIKxaENu"
   },
   "source": [
    "### Check the package versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "Xd-iP9wEaENu"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 2.8.0\n",
      "TFX version: 1.7.1\n",
      "KFP version: 1.8.12\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print('TensorFlow version: {}'.format(tf.__version__))\n",
    "from tfx import v1 as tfx\n",
    "print('TFX version: {}'.format(tfx.__version__))\n",
    "import kfp\n",
    "print('KFP version: {}'.format(kfp.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aDtLdSkvqPHe"
   },
   "source": [
    "### Set up variables\n",
    "\n",
    "You will set up some variables used to customize the pipelines below. Following\n",
    "information is required:\n",
    "\n",
    "* GCP Project id. See\n",
    "[Identifying your project id](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects).\n",
    "* GCP Region to run pipelines. For more information about the regions that\n",
    "Vertex Pipelines is available in, see the\n",
    "[Vertex AI locations guide](https://cloud.google.com/vertex-ai/docs/general/locations#feature-availability).\n",
    "* Google Cloud Storage Bucket to store pipeline outputs.\n",
    "\n",
    "**Enter required values in the cell below before running it**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "EcUseqJaE2XN"
   },
   "outputs": [],
   "source": [
    "GOOGLE_CLOUD_PROJECT = 'qwiklabs-gcp-01-2e305ff9c72b'     # Replace this with your Project-ID\n",
    "GOOGLE_CLOUD_REGION = 'us-central1'      # Replace this with your region\n",
    "GCS_BUCKET_NAME = 'qwiklabs-gcp-01-2e305ff9c72b'          # Replace this with your Cloud Storage bucket\n",
    "\n",
    "if not (GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION and GCS_BUCKET_NAME):\n",
    "    from absl import logging\n",
    "    logging.error('Please set all required parameters.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GAaCPLjgiJrO"
   },
   "source": [
    "Set `gcloud` to use your project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "VkWdxe4TXRHk"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n"
     ]
    }
   ],
   "source": [
    "!gcloud config set project {GOOGLE_CLOUD_PROJECT}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "CPN6UL5CazNy"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PIPELINE_ROOT: gs://qwiklabs-gcp-01-2e305ff9c72b/pipeline_root/penguin-vertex-pipelines\n"
     ]
    }
   ],
   "source": [
    "PIPELINE_NAME = 'penguin-vertex-pipelines'\n",
    "\n",
    "# Path to various pipeline artifact.\n",
    "PIPELINE_ROOT = 'gs://{}/pipeline_root/{}'.format(\n",
    "    GCS_BUCKET_NAME, PIPELINE_NAME)\n",
    "\n",
    "# Paths for users' Python module.\n",
    "MODULE_ROOT = 'gs://{}/pipeline_module/{}'.format(\n",
    "    GCS_BUCKET_NAME, PIPELINE_NAME)\n",
    "\n",
    "# Paths for input data.\n",
    "DATA_ROOT = 'gs://{}/data/{}'.format(GCS_BUCKET_NAME, PIPELINE_NAME)\n",
    "\n",
    "# This is the path where your model will be pushed for serving.\n",
    "SERVING_MODEL_DIR = 'gs://{}/serving_model/{}'.format(\n",
    "    GCS_BUCKET_NAME, PIPELINE_NAME)\n",
    "\n",
    "print('PIPELINE_ROOT: {}'.format(PIPELINE_ROOT))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8F2SRwRLSYGa"
   },
   "source": [
    "### Prepare example data\n",
    "You will use the same\n",
    "[Palmer Penguins dataset](https://allisonhorst.github.io/palmerpenguins/articles/intro.html)\n",
    "as\n",
    "[Simple TFX Pipeline Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple).\n",
    "\n",
    "There are four numeric features in this dataset which were already normalized\n",
    "to have range [0,1]. You will build a classification model which predicts the\n",
    "`species` of penguins."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "11J7XiCq6AFP"
   },
   "source": [
    "You need to make your own copy of the dataset. Because TFX ExampleGen reads\n",
    "inputs from a directory, You need to create a directory and copy dataset to it\n",
    "on GCS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "4fxMs6u86acP"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying gs://download.tensorflow.org/data/palmer_penguins/penguins_processed.csv [Content-Type=application/octet-stream]...\n",
      "/ [1 files][ 25.0 KiB/ 25.0 KiB]                                                \n",
      "Operation completed over 1 objects/25.0 KiB.                                     \n"
     ]
    }
   ],
   "source": [
    "!gcloud storage cp gs://download.tensorflow.org/data/palmer_penguins/penguins_processed.csv {DATA_ROOT}/"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ASpoNmxKSQjI"
   },
   "source": [
    "Take a quick look at the CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "-eSz28UDSnlG"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "species,culmen_length_mm,culmen_depth_mm,flipper_length_mm,body_mass_g\n",
      "0,0.2545454545454545,0.6666666666666666,0.15254237288135594,0.2916666666666667\n",
      "0,0.26909090909090905,0.5119047619047618,0.23728813559322035,0.3055555555555556\n",
      "0,0.29818181818181805,0.5833333333333334,0.3898305084745763,0.1527777777777778\n",
      "0,0.16727272727272732,0.7380952380952381,0.3559322033898305,0.20833333333333334\n",
      "0,0.26181818181818167,0.892857142857143,0.3050847457627119,0.2638888888888889\n",
      "0,0.24727272727272717,0.5595238095238096,0.15254237288135594,0.2569444444444444\n",
      "0,0.25818181818181823,0.773809523809524,0.3898305084745763,0.5486111111111112\n",
      "0,0.32727272727272727,0.5357142857142859,0.1694915254237288,0.1388888888888889\n",
      "0,0.23636363636363636,0.9642857142857142,0.3220338983050847,0.3055555555555556\n"
     ]
    }
   ],
   "source": [
    "# Review the contents of the CSV file\n",
    "# TODO 1: Your code goes here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nH6gizcpSwWV"
   },
   "source": [
    "## Create a pipeline\n",
    "\n",
    "TFX pipelines are defined using Python APIs. You will define a pipeline which\n",
    "consists of three components, CsvExampleGen, Trainer and Pusher. The pipeline\n",
    "and model definition is almost the same as\n",
    "[Simple TFX Pipeline Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple).\n",
    "\n",
    "The only difference is that you don't need to set `metadata_connection_config`\n",
    "which is used to locate\n",
    "[ML Metadata](https://www.tensorflow.org/tfx/guide/mlmd) database. Because\n",
    "Vertex Pipelines uses a managed metadata service, users don't need to care\n",
    "of it, and you don't need to specify the parameter.\n",
    "\n",
    "Before actually define the pipeline, you need to write a model code for the\n",
    "Trainer component first."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lOjDv93eS5xV"
   },
   "source": [
    "### Write model code.\n",
    "\n",
    "You will use the same model code as in the\n",
    "[Simple TFX Pipeline Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "aES7Hv5QTDK3"
   },
   "outputs": [],
   "source": [
    "_trainer_module_file = 'penguin_trainer.py'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "Gnc67uQNTDfW"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing penguin_trainer.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {_trainer_module_file}\n",
    "\n",
    "# Copied from https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple\n",
    "\n",
    "from typing import List\n",
    "from absl import logging\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow_transform.tf_metadata import schema_utils\n",
    "\n",
    "\n",
    "from tfx import v1 as tfx\n",
    "from tfx_bsl.public import tfxio\n",
    "\n",
    "from tensorflow_metadata.proto.v0 import schema_pb2\n",
    "\n",
    "_FEATURE_KEYS = [\n",
    "    'culmen_length_mm', 'culmen_depth_mm', 'flipper_length_mm', 'body_mass_g'\n",
    "]\n",
    "_LABEL_KEY = 'species'\n",
    "\n",
    "_TRAIN_BATCH_SIZE = 20\n",
    "_EVAL_BATCH_SIZE = 10\n",
    "\n",
    "# Since you're not generating or creating a schema, you will instead create\n",
    "# a feature spec.  Since there are a fairly small number of features this is\n",
    "# manageable for this dataset.\n",
    "_FEATURE_SPEC = {\n",
    "    **{\n",
    "        feature: tf.io.FixedLenFeature(shape=[1], dtype=tf.float32)\n",
    "           for feature in _FEATURE_KEYS\n",
    "       },\n",
    "    _LABEL_KEY: tf.io.FixedLenFeature(shape=[1], dtype=tf.int64)\n",
    "}\n",
    "\n",
    "\n",
    "def _input_fn(file_pattern: List[str],\n",
    "              data_accessor: tfx.components.DataAccessor,\n",
    "              schema: schema_pb2.Schema,\n",
    "              batch_size: int) -> tf.data.Dataset:\n",
    "  \"\"\"Generates features and label for training.\n",
    "\n",
    "  Args:\n",
    "    file_pattern: List of paths or patterns of input tfrecord files.\n",
    "    data_accessor: DataAccessor for converting input to RecordBatch.\n",
    "    schema: schema of the input data.\n",
    "    batch_size: representing the number of consecutive elements of returned\n",
    "      dataset to combine in a single batch\n",
    "\n",
    "  Returns:\n",
    "    A dataset that contains (features, indices) tuple where features is a\n",
    "      dictionary of Tensors, and indices is a single Tensor of label indices.\n",
    "  \"\"\"\n",
    "  return data_accessor.tf_dataset_factory(\n",
    "      file_pattern,\n",
    "      tfxio.TensorFlowDatasetOptions(\n",
    "          batch_size=batch_size, label_key=_LABEL_KEY),\n",
    "      schema=schema).repeat()\n",
    "\n",
    "\n",
    "def _make_keras_model() -> tf.keras.Model:\n",
    "  \"\"\"Creates a DNN Keras model for classifying penguin data.\n",
    "\n",
    "  Returns:\n",
    "    A Keras Model.\n",
    "  \"\"\"\n",
    "  # The model below is built with Functional API, please refer to\n",
    "  # https://www.tensorflow.org/guide/keras/overview for all API options.\n",
    "  inputs = [keras.layers.Input(shape=(1,), name=f) for f in _FEATURE_KEYS]\n",
    "  d = keras.layers.concatenate(inputs)\n",
    "  for _ in range(2):\n",
    "    d = keras.layers.Dense(8, activation='relu')(d)\n",
    "  outputs = keras.layers.Dense(3)(d)\n",
    "\n",
    "  model = keras.Model(inputs=inputs, outputs=outputs)\n",
    "  model.compile(\n",
    "      optimizer=keras.optimizers.Adam(1e-2),\n",
    "      loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "      metrics=[keras.metrics.SparseCategoricalAccuracy()])\n",
    "\n",
    "  model.summary(print_fn=logging.info)\n",
    "  return model\n",
    "\n",
    "\n",
    "# TFX Trainer will call this function.\n",
    "def run_fn(fn_args: tfx.components.FnArgs):\n",
    "  \"\"\"Train the model based on given args.\n",
    "\n",
    "  Args:\n",
    "    fn_args: Holds args used to train the model as name/value pairs.\n",
    "  \"\"\"\n",
    "\n",
    "  # This schema is usually either an output of SchemaGen or a manually-curated\n",
    "  # version provided by pipeline author. A schema can also derived from TFT\n",
    "  # graph if a Transform component is used. In the case when either is missing,\n",
    "  # `schema_from_feature_spec` could be used to generate schema from very simple\n",
    "  # feature_spec, but the schema returned would be very primitive.\n",
    "  schema = schema_utils.schema_from_feature_spec(_FEATURE_SPEC)\n",
    "\n",
    "  train_dataset = _input_fn(\n",
    "      fn_args.train_files,\n",
    "      fn_args.data_accessor,\n",
    "      schema,\n",
    "      batch_size=_TRAIN_BATCH_SIZE)\n",
    "  eval_dataset = _input_fn(\n",
    "      fn_args.eval_files,\n",
    "      fn_args.data_accessor,\n",
    "      schema,\n",
    "      batch_size=_EVAL_BATCH_SIZE)\n",
    "\n",
    "  model = _make_keras_model()\n",
    "  model.fit(\n",
    "      train_dataset,\n",
    "      steps_per_epoch=fn_args.train_steps,\n",
    "      validation_data=eval_dataset,\n",
    "      validation_steps=fn_args.eval_steps)\n",
    "\n",
    "  # The result of the training should be saved in `fn_args.serving_model_dir`\n",
    "  # directory.\n",
    "  model.save(fn_args.serving_model_dir, save_format='tf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-LsYx8MpYvPv"
   },
   "source": [
    "Copy the module file to GCS which can be accessed from the pipeline components.\n",
    "Because model training happens on GCP, you need to upload this model definition. \n",
    "\n",
    "Otherwise, you might want to build a container image including the module file\n",
    "and use the image to run the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "rMMs5wuNYAbc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying file://penguin_trainer.py [Content-Type=text/x-python]...\n",
      "/ [1 files][  3.8 KiB/  3.8 KiB]                                                \n",
      "Operation completed over 1 objects/3.8 KiB.                                      \n"
     ]
    }
   ],
   "source": [
    "!gcloud storage cp {_trainer_module_file} {MODULE_ROOT}/"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w3OkNz3gTLwM"
   },
   "source": [
    "### Write a pipeline definition\n",
    "\n",
    "You will define a function to create a TFX pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "M49yYVNBTPd4"
   },
   "outputs": [],
   "source": [
    "# Copied from https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple and\n",
    "# slightly modified because you don't need `metadata_path` argument.\n",
    "\n",
    "def _create_pipeline(pipeline_name: str, pipeline_root: str, data_root: str,\n",
    "                     module_file: str, serving_model_dir: str,\n",
    "                     ) -> tfx.dsl.Pipeline:\n",
    "  \"\"\"Creates a three component penguin pipeline with TFX.\"\"\"\n",
    "  # Brings data into the pipeline.\n",
    "  example_gen = # TODO 2: Your code goes here\n",
    "\n",
    "  # Uses user-provided Python function that trains a model.\n",
    "  trainer = tfx.components.Trainer(\n",
    "      module_file=module_file,\n",
    "      examples=example_gen.outputs['examples'],\n",
    "      train_args=tfx.proto.TrainArgs(num_steps=100),\n",
    "      eval_args=tfx.proto.EvalArgs(num_steps=5))\n",
    "\n",
    "  # Pushes the model to a filesystem destination.\n",
    "  pusher = tfx.components.Pusher(\n",
    "      model=trainer.outputs['model'],\n",
    "      push_destination=tfx.proto.PushDestination(\n",
    "          filesystem=tfx.proto.PushDestination.Filesystem(\n",
    "              base_directory=serving_model_dir)))\n",
    "\n",
    "  # Following three components will be included in the pipeline.\n",
    "  components = [\n",
    "      example_gen,\n",
    "      trainer,\n",
    "      pusher,\n",
    "  ]\n",
    "\n",
    "  return tfx.dsl.Pipeline(\n",
    "      pipeline_name=pipeline_name,\n",
    "      pipeline_root=pipeline_root,\n",
    "      components=components)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mJbq07THU2GV"
   },
   "source": [
    "## Run the pipeline on Vertex Pipelines.\n",
    "\n",
    "You used `LocalDagRunner` which runs on local environment in\n",
    "[Simple TFX Pipeline Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple).\n",
    "TFX provides multiple orchestrators to run your pipeline. In this tutorial you\n",
    "will use the Vertex Pipelines together with the Kubeflow V2 dag runner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7mp0AkmrPdUb"
   },
   "source": [
    "You need to define a runner to actually run the pipeline. You will compile\n",
    "your pipeline into our pipeline definition format using TFX APIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "fAtfOZTYWJu-"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "PIPELINE_DEFINITION_FILE = PIPELINE_NAME + '_pipeline.json'\n",
    "\n",
    "runner = tfx.orchestration.experimental.KubeflowV2DagRunner(\n",
    "    config=tfx.orchestration.experimental.KubeflowV2DagRunnerConfig(),\n",
    "    output_filename=PIPELINE_DEFINITION_FILE)\n",
    "# Following function will write the pipeline definition to PIPELINE_DEFINITION_FILE.\n",
    "_ = runner.run(\n",
    "    _create_pipeline(\n",
    "        pipeline_name=PIPELINE_NAME,\n",
    "        pipeline_root=PIPELINE_ROOT,\n",
    "        data_root=DATA_ROOT,\n",
    "        module_file=os.path.join(MODULE_ROOT, _trainer_module_file),\n",
    "        serving_model_dir=SERVING_MODEL_DIR))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fWyITYSDd8w4"
   },
   "source": [
    "The generated definition file can be submitted using kfp client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "tI71jlEvWMV7"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:Creating PipelineJob\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob created. Resource name: projects/202132981718/locations/us-central1/pipelineJobs/penguin-vertex-pipelines-20220510100554\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:To use this PipelineJob in another session:\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:pipeline_job = aiplatform.PipelineJob.get('projects/202132981718/locations/us-central1/pipelineJobs/penguin-vertex-pipelines-20220510100554')\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/penguin-vertex-pipelines-20220510100554?project=202132981718\n"
     ]
    }
   ],
   "source": [
    "# docs_infra: no_execute\n",
    "from google.cloud import aiplatform\n",
    "from google.cloud.aiplatform import pipeline_jobs\n",
    "import logging\n",
    "logging.getLogger().setLevel(logging.INFO)\n",
    "\n",
    "aiplatform.init(project=GOOGLE_CLOUD_PROJECT, location=GOOGLE_CLOUD_REGION)\n",
    "\n",
    "# Create a job to submit the pipeline\n",
    "job = # TODO 3: Your code goes here\n",
    "job.submit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L3k9f5IVQXcQ"
   },
   "source": [
    "Now you can visit the link in the output above or visit **Vertex AI > Pipelines**\n",
    "in [Google Cloud Console](https://console.cloud.google.com/) to see the\n",
    "progress."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "pknVo1kM2wI2"
   ],
   "name": "Simple TFX Pipeline for Vertex Pipelines",
   "provenance": [],
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m91",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m91"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
